On an inverse Cayley problem

Let G be a group. We will call G a group with Cayley data if we are given all three of the following: the underlying set G; the collection of all Cayley sets of G; and for each Cayley subset S of G, the corresponding Cayley graph Cay(G,S). Is it then possible, from the Cayley data, to reconstruct the binary operation of the group? Is it possible to determine the isomorphism class of the group? We show that there are groups for which the reconstruction of the group operation is not possible, and we call such a group an ambiguous group. We completely characterize ambiguous groups, discuss some properties of the associated Cayley graphs, and we show that the Cayley data for any group does determine the isomorphism class of the group.